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ABSTRACT 

A  procedure  for  optlmiaetion  of  »  oleea  of  biaery  commuaicetioa  ■ye*' 

V  ns  is  pvv'^eated  end  applied.  The  aiessage  set  or  transmitter  input  is  taken 
to  be  a  real-valued  sample  se^[uence  from  a. stochastic  process  with  discrete 
parameter.  The  transmitter  may  be  any  time-varying,  nonlinear  operator 
with  domain  of  the  real  valued  input  and  range  to  the  binary  numbers.  The 
transmission  medium  or  noisy  channel  linking  the  transmitter  and  receiver  is 
to  be  characterised  by  the  conditional  probabilities  of  all  possible  received 
binary  sequences  given  any  transmitted  sequence.  The  receiver  may  be  any 
reaWalued,  time-varying,  no^xiinear  operator  on  the  received  binary  sequences.^ 
Thu  luquii'BBwms  uf  the  matimge  desttwtlewsrie  mturpUfUlU  in  e  systsm  I 
error  criterion,  which  is  taken  as  the  statistical  average  of  a  loss  function, 

4  ,  of  the  difference  between  transmitter  input  and  receiver  output;  the  func 
tion  4  always  taken  as  almost  everywhere  differentiable;  and  additional 
assumptions  such  as  strictly  increasing  for  increasing  magnitude  of  its  argu* 
ment  and  bounded  away  from  minus  infinity,  are  imposed  when  they  lead  to 
interesting  results.  This  restricted  class  of  functions  4  ^  denoted  by 

by  •  . 


The  optimisation  procedure  we  develop  is  much  like  oom  proposed  by 
S.  P.  Lloyd  of  Bell  Telephone  Laboratories  for  use  in  noiseless  ^timum 
quantisation  schemes  and  can  be  so  applied.  Results,  consisting  of  necessary 
conditions  that  the  transmitter  and  receiver  or  quantiser  and  interpolator  must 
satisfy,  can  be  obtained  for  many  members  of  the  family  of  less  functions,  •  , 
referred  to  above.  In  particular,  the  choice  of  a  quadratic  loss,  4(*)  -  > 

is  explored  in  some  detail. 
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y 

The  optimlaatioa  conditioae  obtained  are  diacneeed  and  the  relationahip 

between  an  optimum  communication  eyetem  and  a  delta  modulation  ayetem  in* 

J’/'  /'j  she  VO 

dicated.  ^IPe  a»e  able  te  ehew  that, for  the  quadratic  loee  function  and  any  noiey 
channel^a  delta  modulation  ayetem  ia  an  allowable  repreaentation  of  the  optimum 
binary  -y* —  ^  '  '  '"‘j  '* - 


we  can  prove  t^t  tnesateve  concluaion  alao  obtaina  for  the  claaa  •  of  loaa 
functiona,  4  ,  ^^atiafyi^i 

^  f  *1  I  ^  I  *2  ^ 

Explicit  deaifn  examplea  are\hen  provided  in  addition  to  a  diacuaaion  of  the 
poaaibilitiea  of  automatic  deaiAvia  application  of  digital  computera. 


f  preliminary  reaulta  on  an  extended  digital 


communication  ayatem  model  and 

4  ^ 


areaa  for  further  reaearc^ 


-ill- 
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CONDITIONS  FOR  OPTIMUM  DIGITAL  COMMUNICATION 
WITH  APPLICATION  TO  DSLTA  MODULATION 

by 

Terrance  Fine 

Divieion  of  Engineering  end  Applied  Phyeice 
Cruft  Laboratory,  Harvard  Univereity 
Cambridge,  Maesaehueette 


I.  INTRODUCTION  AND  BACKGROUND 
This  report  attempts  to  present  a  method  of  digital  system  optimiaation 
by  development  and  application.  The  optimum  communication  system,  which 
we  shall  synthesise  by  the  determination  of  necessary  conditions  for  design, 
will  be  seen  to  possess,  tmder  certain  restrictions,  a  representation  as  a 
delta  modulation  system.  Before  proceeding  with  a  more  precise  approach 
to  this  problem,  let  us  establish  a  motivational  background  by  considering 
delta  modulation. 

The  delta  modulation  system  as  proposed  in  1946  is  sketched  in  the 
block  diagram  below.  The  transmitter  possessed  a  feedback  loop  which  inte- 
grated  the  binary  transmitter  output  and  compared  the  result  with  the  current 
input.  According  as  the  input,  I^  ,  at  the  input  time,  t^  ,  was  greater  or 
less  than  the  feedback  value,  L^  ,  the  transmitted  output,  s^  ,  was  in  one 
or  the  other  of  its  two  states.  For  convenience,  the  two  transmitter  output 
states  will  be  taken  as  plus  or  minus  unity,  respectively. 


-I 


The  channel,  which  wae  taken  to  be  noiselees,  provided  the  input  to 
a  receiver  which  then  integrated  this  input  to  generate  its  output.  Thus,  the 
receiver  output  was  similar  to  that  of  the  transmitter  feedback  loop,  and  the 
transmitter  output  served  to  correct  the  receiver  output  insofar  as  it  exceeded 
or  fell  below  the  transmitter  input.  A  common  supplement  to  the  receiver 
indicated  above  was  to  follow  its  output  with  a  band-pass  filter  of  bandwidth 
equal  to  that  of  the  input  message  set. 

The  system  described  above,  taken  together  with  the  one  modification 
of  replacing  the  single  integrator  by  a  double  integrator  to  achieve  a  better 
tracking  response,  represents  the  design  state  of  the  art  until  recently. 
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Optimization  adjuztmentz  with  respect  to  intelligibility  or  noise  power  criteria 
were  made  for  the  most  part  by  experiment.  The  existing  analysis  was  either 
very  qualitative  in  nature  or  attempted  to  determine  properties  of  the  error 
spectrum  for  the  single  or  double  integrating  modulator.  Mention  of  some  of 
the  representative  work  in  this  dbmain  will  be  foimd  in  References  2  through  6. 

Experimental  results  that  were  available  indicated  that  the  delta  modu¬ 
lator  for  speech  inputs  was  inferior  to  pulse  code  modulation  at  moderate  levels 
of  fidelity  and  above.  This  led  us  to  consider  the  possibility  of  generalizing  the 
current  system  and  optimizing  the  more  general  system  assumed  in  order  to 
make  it  k.  c  to  pulse  code  modulation.  The  variations  that  were  attempted 

were  to  remove  the  constraints  that  the  receiver  must  be  like  the  feedback 
loop  and  that  both  of  these  devices  need  be  integrators  or  even  linear  operators. 
We  permitted  the  receiver  and  feedback  loop  to  contain  general  nonlinear 
operators,  including  integrators  as  special  cases.  Then  we  successfully 
attempted  to  derive  the  relations  governing  the  optimum  selection  of  these 
fimctionals  with  respect  to  some  class  of  criteria. 

In  this  report  we  should  like  to  move  the  optimization  problem  one  step 
further  back  by  removing  the  restriction  that  the  transmitter  be  representable 
by  a  feedback  configuration.  We  should  then  like  to  determine  the  conditions 
imder  which  such  a  configuration  will  be  optimum  and  what  its  properties  must 
be,  if  it  is  optimum.  We  shall  also  take  the  communications  channel  to  be 
noisy  and  later  specialize  to  the  noiseless  case.  The  resxilts  we  derive  will 
be  in  the  form  of  necessary  conditions  that  the  components  of  the  optimum 
communication  system  must  satisfy,  and  the  requirements  for  sufficiency 
will  be  only  implicitly  stated. 
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It  should  be  mentioned  that  our  results  were  partially  anticipated  by 
S.  P.  Lloyd  [1]  and  J.  Max  [7]  in  connection  with  the  quantization  problem. 
The  approach  we  employ  was  motivated  and  obtained  independently  from  that 
of  Lloyd,  but  was  first  stated  by  him  in  a  similar  form.  We  have  learned 
from  Lloyd's  unpublished  exposition. 
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II.  SYSTEM  MODEL 

A  precise  description  of  the  binary  communications  system  we  wish  to 
optimize  .jind  discuss  is  summarized  in  the  diagram  below,  and  explicated  in 
this  section.  Possible  extensions  of  this  model  will  be  considered  in  Section 
IX. 


Transmitter 


Chaxmel 


Receiver 
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s  ' 

C 

D 

a 
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»n  =  8(Inl'n-l  > 


«n’  =  S(s„) 


*'n  =  R(«n'l»n-l) 


With  reference  to  the  figure  given  above,  we  take  for  the  input  message 
or  signal  set  (represented  by  I)  a  stochastic  process  with  a  discrete  time 
parameter,  t^  ,  stating  the  time  of  occurrence  of  a  real  valued  input  random 
variable,  I^  .  The  multivariate  probability  distribution  functions  that  describe 
this  stochastic  process  are  assumed  known. 

The  transmitter  operates  on  a  sequence  of  message  inputs  ,  \  ,  to 
produce  a  binary  output  sequence  s^  .  A  superscript  bar  denotes  a  sequence 
extending  into  the  past  with  most  recent  member  indicated  by  the  subscript. 

For  example: 

^n  '  ^  ^n  »  Vl  '  •  '  •  ^n-i+1  J 
where  the  positive  integer  may  be  arbitrarily  large. 

The  requirement  that  a  communication  system  be  binary  implies  that 
the  transmitter  output  sequence,  7^  ,  be  composed  only  of  binary  valued 
elements,  which  for  definiteness  we  take  to  be  plus  or  minus  unity.  It  is 
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important  to  note  that  the  output  binary  sequence  is  of  the  same  length  (same 
subscript)  as  the  input  sequence;  without  loss  of  generality  we  may  neglect 
conuideration  of  the  physical  delay  in  the  transmitter  operation. 

We  represent  the  transmitter  by  some  function  £  with  domain  equal  to 
the  range  of  the  input  and  range  consisting  of  only  plus  or  minus  one,  or 
8^  .  The  function  g  will  depend  upon  the  prior  input  sequence, 
its  explicit  form,  this  being  an  alternative  statement  of  its  functional  depend¬ 
ence  upon  the  entire  past  of  the  input. 

s  =  g(I  IT  .  )  (2) 

The  point  to  be  emphasized  is  that  upon  the  receipt  of  the  input  a  binary 
output  s^  is  generated  in  a  manner  which  depends  upon  the  past  history  of  the 
input  process. 

In  an  actual  system,  the  output  of  the  transmitter  described  above 
would  serve  as  the  input  to  a  pulse  modulator  whose  output  would  in  turn  be 
fed  into  a  communications  channel.  At  the  receiver  a  distorted  version  of  the 
modulator  output,  produced  by  random  and  deterministic  characteristics  of 
the  transmission  medium,  is  transformed  by  the  demodulator  into  another 
binary  sequence,  s^'  .  The  effects  of  the  deterministic  channel  distortions 
may,  in  principle,  be  completely  eliminated,  leaving  only  the  random  distor¬ 
tions  of  the  so-called  noisy  channel.  In  general,  the  terminal  characteristics 
of  such  a  noisy  channel  can  be  described  by  the  conditional  probability  function 
for  an  output  sequence,  s^'  ,  given  an  input  sequence,  s^  ,  and  we  denote 

it  by  P{  I  . 
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The  receiver  operates  on  the  received  sequence,  s^'  ,  to  produce 
real  valued  output  sequence  .  We  assume  that  these  sequences  are  of 
equal  length  or  have  the  same  subscript.  As  before,  there  is  no  loss  of 
generality  attendant  upon  the  neglect  of  physical  time  delays  in  the  actual 
system. 
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III.  STUDY  OBJECTIVE  AND  FORMULATION 

To  design  the  very  general  binary  communication  system  just  described 
so  as  to  optimize  its  performance  in  some  manner,  requires  a  stated  perform¬ 
ance  or  system  error  criterion.  Such  a  criterion  maps  any  transmitter  and 
receiver  functional  pair  into  a  real  number,  and  the  convention  is  adopted  that 
smaller  numbers  represent  preferable  systems.  An  appropriate  and  generally 
useful  class  of  criteria  employs  the  difference  between  system  input  and  output 
as  the  error  variable  and  should  be  used  to  Judge  the  overall  system,  rather 
than  its  individual  components.  The  error  variable,  I  -  R  ,  is  then 
weighted  by  means  of  a  loss  function  ^(I^  -  R^)  chosen  in  accordance  with 
some  preconceived  notions  as  to  what  constitutes  a  significant  characteristic 
of  behavior  (e.  g. ,  we  may  lightly  weight  small  errors  and  '.eavily  penalize 
large  ones). 

By  far  the  most  common  selection  for  f^(x)  is  the  quadratic  function 

2 

•X  and  a  distant  second  might  be  |  x  j  .  However,  for  the  moment  we  take 
^  to  be  almost  everywhere  differentiable  and  bounded  away  from  minus  in¬ 
finity.  Finally,  we  observe  that,  by  the  hypothesis  of  a  random  process  input 
and  a  noisy  channel,  the  input  and  output  and,  therefore,  their  difference,  is 
a  random  variable.  This  requires  us  to  average  ~  over  all  inputs 

and  noisy  channel  outputs  to  obtain  an  overall  error  measure  or  a  mapping  of 
transmitter  and  receiver  independent  of  specific  inputs  and  channel  noise. 

System  error  =  E(^(I  -  R  )}  where  E(  •  )  is  the  statistical  ex- 

n  n 

pectation;  ^  must  be  chosen  so  that  this  quantity  exists  filnitely. 
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For  any  particular  error  criterion  in  the  above  class  we  wish  to  deter¬ 
mine  the  optimum  transmitter  and  receiver,  given  the  statistical  description 
of  the  message  process  and  the  channel.  In  order  to  accomplish  this  objective, 
we  will  first  represent  the  transmitter  and  receiver  more  explicitly. 

In  particular,  we  are  interested  in  the  inverse  image  of  a  plus  one 
transmission  (  =  1  )  with  respect  to  the  mapping  induced  by  the  transmitter 

function,  £  .  From  Eq.  1  of  the  previous  section  the  inverse  image  is  a  set 
of  input  values,  ,  given  by 

I'*'  (T  ,  )  =  f  I  :  g(  I  IT  ,  )  =  +  A 
n'  n-1'  L  n  n  '  n-1  '  / 

The  design  of  a  transmitter  corresponds  to  a  selection  for  the  point  set 

with  the  convention  that  we  transmit  a  plus  one,  if  I  falls  in  ,  and  a  minus 

n  n 

one,  if  I  lies  in  t  ,  the  complement  of  with  respect  to  the  real  line. 

'  n  n  n 

The  receiver  design  consists  of  the  specification  of  a  real  number  for 
each  possible  received  sequence,  s^'  ,  although  only  the  most  recent  mem¬ 
bers  of  the  received  sequence  may  actually  affect  the  choice  of  a  receiver 
output;  we  then  say  the  receiver  has  a  finite  memory.  If  we  assume  that  the 
receiver  possesses  a  finite  memory  of  length  N  ,  then  we  need  only  specify 
2  receiver  outputs,  R^(  s^'  )  .  It  may  be  noted  that  using  a  finite  receiver 
memory  requires  the  specification  of  initial  conditions  to  make  the  problem 
physically  definite,  but  we  need  not  explicitly  evoke  such  a  set  of  conditions 
in  our  present  work.  Finally,  we  take  the  memories  of  the  transmitter  and 
receiver  to  be  of  equal  length  (  equals  N  in  Eq.  1  of  the  second  section). 
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IV.  OPTIMIZATION  PROCEDURE 

In  this  section  we  shall  detearxnine  necessary  conditions  that  and 
the  set  must  satisfy,  if  they  are  to  yield  a  minimum  system  error. 

Lack  of  sufficiency  will  arise,  not  from  the  non-esdstence  of  a  minimum  for 
the  system  error,  but  from  the  possibility  of  several  solutions  to  the  necessary 
conditions.  Sufficiency  is  then  obtained  by  checking  all  of  the  solutions  to  the 
necessary  conditions  by  means  ol  the  value  of  E(|/(  I^  -  R^)  )  ,  or  the  system 
error,  that  they  give  rise  to,  and  then  choosing  that  solution  yielding  the 
minimum  overall  error. 

The  development  of  the  necessary  conditions  for  an  optimum  design 
proceeds  in  three  stages:  (1)  Assume  a  set  of  receiver  outputs 
being  given  and  find  the  optimum  point  set  for  this  choice.  (2)  Assume 
the  point  set  as  known  (and,  therefore,  I*  ,  also)  and  find  the  optimum 
receiver  outputs  corresponding  to  this  transmitter  design.  (3)  Using  the 
interrelated  system  design  equations  of  ( 1  ).and  (  2)  ,  solve  them  simultane¬ 
ously  to  find  the  jointly  optimum  system  design.  This  general  procedure  is 
quite  analogous  to  the  minimisation  of  a  function  of  several  variables  in  the 
calculus;  the  optimum  value  of  each  variable  assuming  all  the  others  constant 
is  found,  and  then  all  of  the  conditions  ars  solved  simultaneously  for  the 
actual  minimum  or  stationary  point. 

The  point  set  is  determined  by  the  requirement  that  the  error  re¬ 
sulting  from  transmission  of  a  plus  one  should  bo  less  than  or  equal  to  that 

I 

resulting  from  transmission  of  a  minus  one  whenever  1^  fblls  in  the  set  . 
The  transmitter  is  aware  of  the  actual  value  of  the  input,  but  possossas  only 
imperfect  knowledge  as  to  the  effect  of  a  transmission  at  the  receiver;  the 
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channel  noise  acts  to  render  the  receiver  output  into  a  random  variable  with 
a  distribution  dependent  upon  the  transmitted  sequence.  Thus,  the  error  we 
wish  to  minimize  is  an  error  averaged  over  all  of  the  possible,  initially  given, 
receiver  outputs.  Our  discussion  is  then  summarized  by 

— E[^(I„  -  R„(  V)  )|7„*]<  E[/(I„  -  )|  VI  (1) 

where 

*n  ~  ~  ^  ’  *n-l  ^  ’  *n  “  ^*n  ”  ^  *  *n-l  ^ 

We  digress  briefly  to  introduce  a  representation  for  the  set  as  a 

union  or  sum  (as  the  components  are  disjoint)  of  closed  intervals,  with  closure 

being  guaranteed  by  the  presence  of  the  equality  sign  in  the  above  inequality. 

0  =  U  [  L.,-  ,  L  •  ]  with  L  .  ^  L  .  (2) 

n  '-r  '■  ox  ’  VI  •'  ox  VI  •  '  ' 


We  should  like  to  find  a  necessary  condition  that  the  end  points  of  the 
intervals  in  the  above  imion  must  satisfy.  Observe  that  ,  being  the  com¬ 
plement  of  ,  consists  of  the  open  intervals  adjacent  to  the  closed  intervals 
of  .  Thus,  for  inputs  slightly  below  the  left  end  point  of  any  interval  in 
or  slightly  above  the  right  end  point  of  any  interval  in  we  have  that  a  minus 
one  transmission  is  optimum  or  that  the  direction  of  the  inequality  in  Eq.  1  is 
reversed.  The  assumed  continuity  of  ^  and  the  reversal  of  the  ineqxxality  as 
the  input  is  varied  about  the  end  points  assure  us  that  the  end  points  of  the  in¬ 
tervals  in  either  or  (sind,  excepting  +  oo  ,  they  are  the  same)  must 
be  those  real  input  values  for  which  equality  obtains  in  Eq.  1  .'  More  concisely. 
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Not  all  of  the  real  roots  of  the  above  eqxiation  necessarily  correspond  to 
the  end  points  of  an  interval,  for  only  those  points  of  equality  that  mark  a 
transition  in  the  direction  of  the  inequality  1  are  of  significance;  this  is;  then, 
only  a  necessary  condition  for  the  optimxun  transmitter  given  the  receiver.  If 
both  sides  of  the  inequality  1  evaluated  at  the  point  L  ,  which  is  a  solution 
of  Eq.  3,  do  not  have  the  same  slope,  or  derivative  with  respect  to  at  the 
point  L  ,  then  the  direction  of  the  inequality  1  imdergoes  a  reversal  at  the 
point  where  eqtials  L  .  Thus,  a  condition  that  guarantees  a  transition 
in  the  inequality  for  a  root  L  of  Eq.  3  is 

We  should  mention  that  the  new  condition  provided  by  Eq.  3  is  sufficient  to 
guarantee  a  change  in  the  direction  of  the  inequality  1,  but  not  necessary; 
equality  may  obtain  and  L  still  be  a  point  of  transition,  because  it  is  also  a 
point  of  inflection.  We  shall  discuss  the  properties  of  these  results  more  fully 
in  the  next  section. 

If  we  now  fix  the  transmitter  design  and  turn  to  the  receiver,  we  note 
that  the  receiver  knowing  the  input  statistics,  the  transmitter  design,  the 
channel  noise  statistics,  and  the  received  sequence  can  determine  the  diltri- 
bution  function  of  the  input  conditional  upon  the  received  binary  sequence.  The 
receiver  must  then  process  this  distribution  fimction  to  yield  that  niunber 
which  results  in  a  minimum  system  error.  That  is,  we  must: 

minimize  E  (^(  1^^  -  |  s^^' )  (5) 

^n 

The  conditional  expectation  has  an  integral  representation  employing  the 
conditional  distribution  function  for  the  input  given  the  received  sequence. 
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A  necessary  condition  that  the  number  must  satisfy  to  minimize  the 
integral  can  be  obtained  from  the  calculus  and  is: 

00 

5I-  .  (6) 

-00 

Interchanging  orders  of  differentiation  and  integration,  we  have: 

00 

J,r'(x-R^)dP[x>I^|7^']=  0  .  (7) 

-00 

The  above  equation  rewritten  more  compactly  yields  the  necessary  condition 
that  determines  the  optimvim  receiver  for  a  fixed  transmitter. 

The  final  step  in  an  actual  optimization  problem  would  be  to  search 
through  the  solutions  of  both  of  the  derived  necessary  conditions  Eqs.  3  and  8 
to  find  that  pair  resvilting  in  the  absolute  minimum  system  error.  It  may  be 
possible  to  solve  this  problem  in  a  closed  analytic  fashion,  but  this  is  generally 
the  exception.  More  commonly,  we  would  be  driven  to  some  iterative  procedure. 

In  the  next  two  sections  we  shall  indicate  some  conditions  under  which 
there  is  only  one  real  L  solution  to  £q.  3;  is  then  a  single  semi-infinite 
interval.  In  such  a  practically  important  case,  the  iterative  procedure  might 
proceed  by  first  assviming  a  trial  value  of  L  and  using  it  in  Eq.  8  to  determine 
the  various  receiver  outputs.  With  the  calculated  receiver  outputs  we  can  turn 
to  Eq.  3  and  determine  a  check  value  of  L  .  If  the  check  value  is  lower  than  the 
trial  value  of  L  ,  then  the  next  trial  value  should  be  decreased  from  its  original 
magnitude  and  vice  versa.  This  particular  interative  procedure  is  only  meant  as 
an  illustration  and  is  only  known  to  be  applicable  in  the  examples  to  be  dis¬ 
cussed  shortly  where  L  is  unique. 
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V.  IMPLICATIONS  OF  THE  DERIVED  NECESSARY  CONDITIONS 
Equation  3  for  the  tranamitter  design  depends  upon  the  actual  input 
process  only  insofar  as  !■  concerned  and  in  no  other  way.  We  have  the 

direct  conclusion  that  the  end  points  of  the  intervals  making  up  depend  only 
upon  s^_^  ;  therefore,  is  a  functional  only  of  >  and,  finally,  g  is 

also  only  dependent  upon  the  past  of  the  transmitted  sequence.  This  argument 
produces  the  interestuig  conclusion  that  the  optimum  transmitter  does  not 
utilize  the  fully  available  past  of  the  input,  I^_^  ,  but  only  the  transmitter 
output  derived  from  that  input  sequence,  .' 

If  for  the  loss  function,  4  >  use  the  very  common  quadratic  func¬ 
tion,  we  can  rewrite  Eq.  3  by  utilizing  the  observation  that  in  Eq.  3  L  is  just  a 
real  nvimber  and  can  commute  with  the  conditional  expectation  operator.  This 
leads  to 


2  I  T  +  ■ 


2 

Eliminating  the  L  terms  and  solving  the  resulting  linear  equation  leads  to 
the  following  explicit  expression  for  L  : 


L(  s 


n- 1 


e[rJV1-e[rJv1 


(2) 


The  significance  of  Eq.  2  is  that  for  any  assumed  receiver  outputs  there  is  one 
and  only  one  solution  to  Eq.  3;  is  just  a  semi-infinite  interval. 

The  above  conclusions  for  the  quadratic  loss  function  lead  us  to  the 
realization  that  an  alternative  block  diagram  for  our  binary  communications 
system  can  be  given  utilizing  a  feedback  loop. 
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If  I  falls  in  the  semi -infinite  interval  with  lower  end  point  of  L  ,  then  D 
is  positive,  s^  is  plus  one,  and  we  are  classifying  the  inputs  by  means  of  the 
proper  form  of  point  set,  The  feedback  loop  provides  precisely  the 

s^_j  needed  to  generate  correctly  the  lower  end  point  of 

This  feedback  device  is  the  generalization  of  the  delta  modulation  sys¬ 
tem  investigated  by  others,  and  ourpresent  work  shows  that  the  delta  modula¬ 
tion  configuration  leads  to  a  quadratically  optimal,  binary  communication  sys¬ 
tem  in  the  presence  of  an  arbitrarily  noisy  channel Whether  or  not  t^s  result 
(of  a  unique  solution  to  Eq.  3  of  the  previous  section),  holds  for  a  variety  of  loss 
functions  is  at  present  unknown,  but  an  affirmative  answer  is  expected. 

The  necessary  condition  for  an  optimal  receiver  given  by  Eq.  8  of  the 
previous  section  is  a  reasonable  one.  The  information  at  the  receiver  concern¬ 
ing  transmitter  input  and  channel  noise  statistics  as  well  as  the  transmitter  de¬ 
sign  and  the  received  binary  sequence  enable  it  to  determine  the  conditional 
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probability  of  th«  input  given  all  of  this  information.  The  receiver  then  selects 
the  "beet”  single  estimate  ol  the  input  from  this  distribution  and  the  given  lose 
function.  If  the  lose  function  ia  quadratic,  one  can  readily  show  that  the  receiv¬ 
er  operates  by  taking  the  conditional  expectation  of  the  input,  given  the  received 
sequence  as  its  output;  the  familiar  mean  as  the  solution  to  the  minimum  vari¬ 
ance  problem.  In  a  similar  vein,  if  the  loss  function  is  taken  as  the  absolute 
value  of  the  error  variable,  then  we  find  that  the  optimum  receiver  has  as  its 
output  the  median  of  the  conditional  distribution  of  the  system  inpxit  given  the 
received  binary  sequence.  This  result  is  also  well  known. 
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VI.  NOISELESS  CHANNEL 

An  important  ■pacialiaation  of  our  result!  is  to  the  case  of  the  noise¬ 
less  channel  or  the  channel  in  which  the  received  binary  sequence  is  identical 
with  the  transnnitted  sequence,  7^  .  The  transmitter  design  relation  can  now 
be  appreciably  simplified  by  the  observation  that  the  receiver  output  given  the 
transmitted  sequence  is  a  degenerate  raxidom  variable  or  constant,  and,  there¬ 
fore,  the  expectation  is  superfluous.  Equation  3  of  the  optimization  section 
reduces  to: 

If  we  generalize  from  the  quadratic  and  absolute  value  loss  functions  to  a  class 
oi  4  i  4  >  which  satisfies 

4  a.  e.  differentiable  ;  (  z  j  |  >  (  z^  (  4=^ /(  z^  )  >  ^(  Zj, )  >  -  oo  (2) 
then  the  unique  solution  of  Eq.  1  becomes: 

‘'=K<V'  +  K<v>  =  ^<vi)  • 

Equation  3  can  be  verified  by  use  of  Eq.  2  and  substitution  of  Eq.  3  in  Eq.  1 ; 
uniqueness  follows  from  the  strict  inequality  in  Eq.  2  . 

The  essential  point  is  that,  for  the  noiseless  channel  and  an  important 
class  of  loss  functions,  classification  of  the  input,  at  a  particular  time,  is 
only  based  upon  whether  it  is  larger  or  smaller  than  a  particular  number,  L . 
This  is  the  same  conclusion  that  we  reached  for  the  quadratic  criterion  and 
the  noisy  channel,  and,  therefore,  the  representation  of  the  optimum  system 
as  a  delta  modulation  system  is  again  possible. 

The  specialization  of  the  receiver  relation  is  trivial  and  is  simply: 
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VII.  CALCULATED  DESIGN  EXAMPLES  ( NOISELESS) 

Equations  3  and  4  of  Section  VI  provide  us  with  conditions  that  must  be 
satisfied  by  the  optimvim  transmitter  and  receiver,  respectively.  These  equa¬ 
tions  are  not  difficult  to  implement,  if  we  are  provided  with  an  analytically 
tractable  conditional  distribution  or  density  function  for  the  input  process. 
Uxifortunately,  it  is  far  from  common  that  the  multivariable  distribution  func¬ 
tions  for  an  input  stochastic  process  will  be  given  in  terms  of  finite  combina¬ 
tions  of  elementary  functions,  and  this  makes  calculations  with  such  functions 
laborious.  Two  exceptions  to  this  dictum  will  be  examined  below,  and  they 

represent  opposite  ends  of  the  range  for  dependence  between  members  of  a 

1  2 
sequence.  The  loss  function  p(x)  will  be  taken  to  be  quadratic  or  x 

Independence 

We  first  assume  that  the  input  stochastic  process  consists  of  a  sequence 
of  independent,  identically  distributed  random  variables  with  common  density 
function  p(I)  .  Then  from  Eq. 4  and  the  quadratic  loss  we  have,  after  differ¬ 
entiation  and  separation  of  terms 

=  ;  ®n< 'n  ‘ I ‘n ‘ 

The  receiver  output  shovild  be  the  conditional  mean  or  the  mean  of  the  input 
distribution  as  known  at  the  receiver.  The  mutual  independence  of  members 
of  the  input  sequence  and  the  given  density  function  allow  us  to  rewrite  Eq.  1 
by  employing  the  integral  representation  for  the  expectation 
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R  (•  s  1 )  s 


n'  n 


00 

J  xp(x)  dx 

K _ 

00 

J  p(x)dx 


J  xp  { X )  dx 
-00 


(2) 


n 

I 


-00 


P(x)dx 


,  ii  the  as  yet  unknown  position  of  the  transmitter  feedback  loop 
output,  and  it  can  be  foxmd  from  Eq.  2  above  and  Eq.  3  of  Section  VI  as  follows: 


I 


xp  ( X  )  dx 


f  xp(x)  dx 


L  -00 

=  ^ -  +  T: -  •  (3) 

00  n 

j*  p  ( X  )  dx  J  p  ( x )  dx 

L  -oo 

n 

If  we  have  a  specific  p  ( I )  in  mind,  we  may  proceed  to  solve  the 
transcendental  equation  given  by  Eq.  3  for  and  then  use  in  Eq.  2  to 
determine  the  two  receiver  outputs.  For  example,  if  p(I)  is  the  exponential 
distribution  for  positive  J,  ,  the  integration  can  be  carried  out,  and  Eq.  3 
yields : 


exp {-L)s  1  -  or  L  =  1 .  6  . 
n  *  n 

Having  found  the  unique  setting  of  the  feedback  loop,  we  may  return  to  Eq.  2 
and  calculate  the  two  receiver  outputs 
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Further  calculations  are  straightforward  and  we  siunmarize  their  results: 

Prob  (  8^  =  1  )  =  .  2  ;  var  =  1  i  E  (  ( -  R„  )^ )  =  •  36  . 

This  completes  the  design  of  the  optimum  system  for  such  an  input  process, 
and  describes  its  performance. 


Complete  Dependence 

The  case  of  complete  dependence  in  chains  of  length  M  and  inde¬ 
pendence  between  successive  chains  can  be  treated  fairly  directly,  because 
of  the  relative  simplicity  of  the  resulting  distribution  functions.  Ve  assume 
that  through  are  equal  to  some  random  variable  I  with  density 
function  p(I)  .  Then  the  transmitter  and  receiver  design  for  the  first  mem¬ 
ber  of  the  sequence,  1^  ,  are  precisely  as  given  in  Eqs.  2  and  3  above.  As 
an  illustration  of  how  the  problem  proceeds,  we  give  the  design  equations  for 
the  system  concerning  I2  and  assuming  that  1  •  Differences  occur  in 

the  interval  of  integration  and  in  the  denominator,  due  to  the  change  in  the 
conditional  density  function 


Pd,) 


pdzi-i  =  U=  \ 


00 

J  Pd)  dl 

L, 


if  I2  >  Lj 


(4) 


if  l2<  Lj 


The  design  equations  then  become 
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The  process  just  indicated  is  then  continued  until  the  siep*  after  which 

it  recycles. 

It  is  possible  to  apply  Eq.  5  to  the  exponential  distribution  discussed  in 

the  case  of  independence.  If  one  does  so,  one  straightforwardly  finds  after 

integration  and  the  solution  of  a  simple  transcendental  equation: L^Sj^  =  1)  ±  3.20; 

R2(«i  =  «2  =  ^  f  R2(»i  =  ’»2  =  1  )  =  2.20  ;  Prob (S2  =  1 1 Sj=  1)  =  .203; 

2 

var  I2  ~  1  t  £!  (  ( I2  ~  ^2  ^  <  *1  ~  ^  ^  ~  notice  that  the  second  step 

provides  little  improvement  in  perfoamiance  over  the  first  step,  under  the  hy¬ 
pothesis  that  ■2  =  1  •  continuation  of  this  example  to  investigate  the  other 

alternatives  is  simple,  but  involves  repetitive  elementary  calculations. 

As  another  illustration  we  might  consider  that  the  density  function, 
p  ( I )  ,  was  vmiform  over  some  range  H  .  We  would  then  find  that  L12  occurred 
at  the  midpoint  of  the  range,  and  the  two  receiver  outputs,  R2  (*2  ~  ^  ^ 

Rl  (  Si  =  -  1 )  ,  occurred  at  the  midpoints  of  the  right  and  left  halves  or  at 
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the  3/1  and  1/4  points  of  the  full  range,  respectively.  The  transmitter  con¬ 
tinues  operation  by  successively  halving  the  remaining  range  into  which  1_  is 
known  to  have  ihllen,  and  the  receiver  output  is  taken  at  the  midpoint  of  the 
reduced  range.  Finally,  the  digit  asserts  that  is  in  some  interval  of 

length  H/2^  and  that  the  receiver  output  is  at  the  midpoint  of  this  interval. 

The  system  performance  may  then  be  readily  calculated  at  each  step,  and  we 
summarise  the  results  that  obtain  Just  after  the  J^b  binary  digit  has  been  re¬ 
ceived;  var  Ij  s  H^/l2  ;  E  (  ( Ij  -  Rj  )^  |  Sj  )  *  /  12.  ;  Prob(  Sj^2  }7j)  =l/2. 

These  results  for  J  equal  to  unity  also  provide  the  solution  to  the  case  of 
independence  between  inputs  when  the  input  distribution  is  uniform. 


Pd) 


H 


.H  Ri(sj  =  -1)  Lj  Rl(«l»l) 
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The  case  of  dependence  that  we  have  discussed  above  corresponds  to  a 
process  of  M-bit  pulse  code  modulation  (PCM)  .  The  input  value  1^  is 
represented  by  a  binary  sequence  M  in  length,  from  which  its  magnitude  can 
be  reconstructed.  This  particular  quantization  procedure  is  sometimes  re¬ 
ferred  to  as  programmed  PCM  . 
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VUI.  COMPLEMENTS  FOR  DELTA  MODULATION 
The  reeiilte  deicribed  in  the  previous  sections  have  been  examined  in 
greater  detail  than  hitherto  presented,  particularly  with  regard  to  the  proper¬ 
ties  of  a  delta  modulation  representation  and  a  noiseless  channel.  In  this  sec¬ 
tion  we  should  like  to  discuss  briefly  two  results  that  follow  from  our  analysis, 
as  well  as  to  indicate  our  application  of  a  digital  computer  to  obtain  additional 
conclusions. 

We  notice  from  Eq.  3  of  Section  VI  that  the  optimxim  transmitter  feed¬ 
back  loop  is  linearly  related  to  the  chosen  receiver.  If  for  various  reasons, 
including  engineering  practicality,  we  wish  to  select  a  linear  operator  on  the 
received  binary  sequence,  or,  alternatively  stated,  a  linear  time-varying  sys¬ 
tem,  for  the  receiver,  then  we  can  show  that  the  optimum  transmitter  feedback 
loop  should  be  chosen  to  be  the  same  operator  restricted  to  the  past  of  the 
received  sequence.  To  prove  this  statement,  take  the  following  representation 
for  the  receiver: 


a  s 
nm  m 


(1) 


m 


Now  the  expression  for  the  optimum  transmitter  feedback  loop  operator, 
^n^*n-l  ^  given  by: 

2^n< -n-l )  =  «n<  'n  =  '  »  Vl  >  ^  ^n<  -n  =  ■  '  »  Vl  )  ' 

Substitution  of  Eq.  1  into  Eq.  2  yields  the  optimum  solution  for  L^. 

n-1 


(2) 


*nm  *m 


(3) 


m 
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Equation  3  proves  our  conjecture  concerning  the  relation  of  the  optiznxun  trans¬ 
mitter  to  the  receiver  constrained  to  be  a  linear  operator. 

Equation  3  also  provides  a  justification,  in  part,  for  the  assumption 
made  by  the  early  designers  of  delta  modulation  systems  that  the  feedback  loop 
element  should  be  the  same  as  the  receiver.  The  subsequent  refinement  by  the 
addition  of  a  band-pass  filter  at  the  output  of  the  receiver  was  motivated  by  an 
attempt  to  interpolate  between  samples  and  does  not  appear  in  our  discrete  time 
analysis,  which  is  concerned  only  with  the  samples  themselves. 

A  further  result  is  an  approximation  to  the  transmitter  design  equation 
based  upon  the  hypothesis  that  at  a  given  time  a  positive  transmission  is  as 
likely  as  a  negative  one.  This  assumption  implies  that  we  may  rewrite  Eq.  2 
above  as  follows: 


If,  in  addition,  we  assume  a  quadratic  loss  function  so  that  the  optimum  re¬ 
ceiver  is  a  conditional  expectation  itself,  then  Eq.  4  becomes 


The  significance  of  Eq.  5  is  that  the  transmitter  design  equation  no 
longer  depends  upon  the  receiver  design,  and  we  may  solve  directly  rather 
than  iteratively.  Thus,  by  assuming  a  property  of  the  resulting  optimum  sys¬ 
tem,  which  is  known  not  to  be  viniversally  valid,  we  have  simplified  the  analy¬ 
sis  leading  to  the  optimum  design.  This  approximation  is  essentially  that  we 
will  transmit  at  the  maximum  information  rate  with  an  optimum  binary  com¬ 
munication  system  as  we  have  defined  it;  this  statement,  however,  is  only 
occasionally  correct,  and  its  truth  depends  upon  the  nature  of  the  input  process. 
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l%e  maximum  information  rate  approximation  referred  to  above  im¬ 
plies  a  very  special  structure  for  the  input  stochastic  process,  if  it  is  to  hold 
exactly.  In  detail  it  requires  that  at  the  time  in  question  the  conditional  mean 
of  the  input  distribution  be  equal  to  the  conditional  median  of  the  input  distri¬ 
bution.  If  the  probability  distribution  function  for  the  input  at  a  given  time  con 
ditional  upon  the  past  of  the  transmitted  binary  sequence  is  symmetric,  then 
the  conditions  are  met  and  the  approximation  is  exact.  An  example  of  such  a 
process  would  be  the  uniformly  distributed  dependent  case  of  Section  VII  or 
any  symmetrically  distributed  input  process  of  independent  random  variables. 
The  multivariate  normal  process  does  not  satisfy  the  conditions  we  have 
suggested  as  establishing  the  exact  validity  of  our  suggested  approximation. 
However,  computer  studies  of  selected  exponentially  correlated  and  band 
limited  normal  processes  indicated  that  the  error  made  in  applying  our  approx 

imation  to  the  determination  of  h  was  less  than  15%. 

n 

Computer  studies  based  upon  the  results  for  a  noiseless  channel  and 
quadratic  loss  function  have  been  carried  out  with  the  following  objectives: 

(1)  Determine  the  feasibility  of  the  iteration  scheme  outlined  earlier.  (2) 

Judge  the  accuracy  of  the  maximum  information  rate  approximation  discussed 
above.  (3)  Provide  an  automatic  design  procedure  for  optimtun  binary  com- 
mimication  systems  given  only  input  data  and  not  input  statistics.  (4)  Effect 
a  simulation  procedure  for  use  in  investigating  system  performance  when  the 
input  statistics  preclude  reasonable  analytic  solution.  Valuable  conclusions 
were  obtained  in  all  of  the  above-mentioned  areas  but  the  last.  A  procedure 
is  available,  but  the  results  of  its  application  are  inconclusive. 
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The  maximum  information  rate  approximation  referred  to  above  im- 
pliee  a  very  special  structure  for  the  input  stochastic  process,  if  it  is  to  hold 
exactly.  In  detail  it  requires  that  at  the  time  in  question  the  conditional  mean 
of  the  input  distribution  be  equal  to  the  conditional  median  of  the  input  distri¬ 
bution.  If  the  probability  distribution  function  for  the  input  at  a  given  time  con¬ 
ditional  upon  the  past  of  the  transmitted  binary  sequence  is  symmetric,  then 
the  conditions  are  met  and  the  approximation  is  exact.  An  example  of  such  a 
process  would  be  the  uniformly  distributed  dependent  case  of  Section  VII  or 
any  symmetrically  distributed  input  process  of  independent  random  variables. 
The  multivariate  normal  process  does  not  satisfy  the  conditions  we  have 
suggested  as  establishing  the  exact  validity  of  our  suggested  approximation. 
However,  computer  studies  of  selected  exponentially  correlated  and  band 
limited  normal  processes  indicated  that  the  error  made  in  applying  our  approx¬ 
imation  to  the  determination  of  L  was  less  than  15%. 

n 

Computer  studies  based  upon  the  results  for  a  noiseless  channel  and 
quadratic  loss  function  have  been  carried  out  with  the  following  objectives: 

(1)  Determine  the  feasibility  of  the  iteration  scheme  outlined  earlier.  (2) 

Judge  the  accuracy  of  the  maximum  information  rate  approximation  discussed 
above.  (3)  Provide  an  automatic  design  procedure  for  optimum  binary  com¬ 
munication  systems  given  only  input  data  and  not  input  statistics.  (4)  Effect 
a  simulation  procedure  for  use  in  investigating  system  performance  when  the 
input  statistics  preclude  reasonable  analytic  solution.  Valuable  conclusions 
were  obtained  in  all  of  the  above-mentioned  areas  but  the  last.  A  procedure 
is  available,  but  the  results  of  its  application  are  inconclusive. 
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The  iteration  scheme  proved  practicable  on  an  IBM  7090  when  used  in 
conjunction  with  appropriate  statistical  estimators.  Input  information  was 
provided  in  the  form  of  independent  sequences  of  fixed  length  and  arithmetic 
average  estimators  were  used  to  provide  the  needed  conditional  expectations. 
The  iteration  process  will  always  converge  in  a  finite  nvunber  of  steps,  the 
number  of  steps  being  bounded  by  the  total  number  of  independent  sequences 
providing  the  input  information.  This  overall  procedure  yields  a  valuable 
means  of  automatically  designing  an  optimum  binary  communication  system 
directly  from  measurements  on  the  message  or  input  process.  The  appendix 
contains  an  illustrative  example  of  the  calculational  process  alluded  to  above. 

It  was  also  possible  to  estimate  the  optimum  system  for  an  input 
process  with  known  statistics,  such  as  multivariate  normal,  by  first  simu¬ 
lating  the  input  process  and  then  proceeding  as  above.  The  optimum  system 
could  then  be  tested  and  its  performance  evaluated  by  operating  the  simvilated 
system  on  additional  members  of  the  message  set  and  comparing  the  system 
output  with  the  known  input.  However,  the  conclusions  so  reached  have  not 
been  particularly  fruitful.  It  is  intended  to  pursue  this  work  somewhat  further 
by  processing  recorded  quantized  speech  provided  by  the  Bell  Telephone 
Laboratories  in  order  to  determine  the  merits  of  the  system  we  have  devised 
with  respect  to  potential  speech  communication  applications;  this  investigation 
will  probably  be  carried  out  for  the  quadratically  optimal  system.  We  hope  that 
the  practicality  of  the  simulation  and  automatic  design  procedure  we  have 
developed  for  the  design  of  optimum  binary  communication  systems  will  serve 
as  an  inducement  to  application  even  in  those  instances  where  a  fully  analytic 
solution  is  very  difficult. 
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IX.  EXTENSIONS  TO  THE  BINARY  MODEL 
The  bulk  of  thie  report  hae  been  devoted  to  a  diecueeion  of  what  we 
refer  to  a«  a  binary  commiinication  eyetem,  and  most  of  our  work  hae  been  in 
this  direction.  However,  it  is  possible  to  treat  the  more  complex  model  of  an 
M'^ary  digital  system  employing  reconstruction  delay  by  the  methods  we  have 
developed,  and  we  shall  indicate  the  procedure  in  this  section.  It  should  be 
stated  at  the  outset  that  using  our  procedure  to  determine  necessary  conditions 
no  longer  seems  as  profitable;  generalizing  the  system  model  seems  to  have 
weakened  the  derived  necessary  conditions. 

Referring  to  the  block  diagram  of  the  general  communication  system 
provided  in  the  first  section,  we  write  the  system  equations  as  follows. 

•n=8(InlVl)  V  =  ;  Rn=RCn'l^'n-l) 

minimize  E  ( ( I^  -  ) 

“n»  *^n+k 

We  must  now  remember  that  £  represents  an  M~ary  valued  symbol  rather 
than  just  a  binary  valued  one.  The  only  change  in  the  form  of  the  above 
equations  from  their  previous  values  occurs  in  the  error  criteria  where  we 
assume  a  delay  of  k  inserted  at  the  receiver. 

The  error  criterion,  E(^(I^  -  >  involves  an  averaging 

over  all  of  the  possible  receiver  outputs  that  can  occur  at  time  t^^j^  given 
that  has  been  the  system  input.  The  presence  of  a  noisy  channel  gen¬ 
erally  implies  that  this  average  is  to  be  taken  over  all  possible  receiver  out¬ 
puts  at  time  t^^j^  ,  and  assuming  that  the  sequence  I^  has  length  (i  +  1 )  , 
we  see  that  there  will  be  possible  different  receiver  outputs  to  be 

averaged  over.  If  the  channel  had  been  noiseless,  then  the  number  of  receiver 
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outputs  to  be  averaged  over  would  only  have  been  M  ,  which  for  zero  k 
reduces  to  unity  as  expected. 

The  formal  optimization  procedure  is  now  very  similar  to  that  devel¬ 
oped  in  the  third  and  fourth  sections  for  the  binary  system  operating  without 
intentional  delay.  The  essential  distinction  lies  in  the  greatly  increased  num¬ 
ber  of  akematives  now  available  for  muttial  comparison.  At  t^  we  must  decide 
which  of  M  states  to  select  for  transmission  or  equivalently  determined  the 
M  disjoint  and  exhaustive  subsets  of  the  real  line,  ij^(j)(j=l,...,M)  , 
where  fjj(j)  is  given  by: 

Vk>l\-  ’n'Jl  «  ,2, 

less  j  J 

Solutions  for  Eq.  2  will  then  yield  the  optimum  transmitter  to  be  used  in  asso¬ 
ciation  with  a  given  receiver.  Arguments  similar  to  those  presented  in  the 
binary  case  indicate  that  if  we  represent  j  )  as  a  union  of  disjoint  inter¬ 
vals,  then  the  endpoints  of  these  intervals,  ip)  >  must  satisfy. 


6  pyi)E[.<(L,(J,y-V,,)  I 


(3) 


•EU<VJ>‘p'- Vk>|l».-„*Pl  ■ 

Unfortunately,  Eq.  3  is  only  a  weak  and  uninformative  necessary  condition  for 
the  determination  of  fj^(j)  • 

The  optimum  receiver  determination  proceeds  precisely  as  it  did 
earlier.  The  transmitter  is  assumed  given,  and  we  find  by  strict  analogy  with 
our  earlier  results  that 
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Fixuilly,  we  are  required  to  conelder  the  results  of  Eqs,  3  and  4  simultaneously, 
and  select  that  pair  resulting  in  the  minimum  value  of  the  error  term  stated 
in  Eq.  1  . 

The  results  stated  above  will  become  the  results  given  in  the  previous 
section  when  we  replace  M  by  2  and  thereby  provide  a  more  general  formu¬ 
lation  of  the  digital  communication  problem.  A  special  case  of  some  Interest 
occurs  when  we  consider  the  possibility  of  a  noiseless  channel  and  no  delay . 
Equations  3  and  4  then  become: 

^  ^  for  some  (p  /  j)  ,  (3') 

\)|  =  0  .  (4') 

Observe  that,  if  we  take  ^  to  be  in  4  or  such  that 

4  4  (>2  )  ^  1^1 1  ^  1^2^  * 

then  there  will  be  a  unique  solution  to  Eq.  3*  ,  for  every  ^  given  by 

2L^(j,p)  =  R^(j)  +  R^(p)  (5) 

where 

Vj,p)  = 

and  is  unique. 

The  result  expressed  by  Eq.  5  contains  much  irrelevant  information. 

If  we  return  to  the  original  inequality  formulation  as  given  by  Eq.  2,  we  can 

select  those  solutions  of  Eq.  5  which  are  important.  For  convenience  we 

order  the  receiver  outputs  as  follows: 

( j  )  >  ( i)  . if  and  onlv  if  j  >  i  .  (6) 

Using  Eqs.  5  and  6  in  Eq.  2  leads  to  the  conclusion  that  J  )  need  only  be 

a  simple  interval  located  between  i^(J  -  1)  below  it,  and  ^,^0  i)  nbove  it. 
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and  Ijj(j)  contains  the  point  •  The  upper  end pdint  of  the  interval 

ln(J)  bisects  the  interval  between  <^nd  M  >  while  the  lower 

end  point  of  Just  the  upper  endpoint  of  f,j(J  ~  M  i  the  only  exceptions 

to  these  rules  are  that  the  lower  end  point  of  ( 1  )  is  minus  itifinity  and  the 
upper  end  point  of  is  plus  infinity. 

-00  < - ill)  4 - ^\4 - ^  f  (2)4 - ^  i  (3)  i - ^  +  00 

n  in  in 

- 1 - 1 - - 1 - 1 - 

RJl)  LJ1,2)  R^(2)  L^{2,3>  RJ3)  ” 

TYPICAL  TRANSMITTER  ASSIGNMENT 
(M  =  3) 

Essentially, these  conclusions  were  obtained  by  Lloyd  [1]  under  the 
restriction  that  ^  be  convex  and  even,  the  channel  be  noiseless,  and  there 
be  no  reconstruction  delay.  Insofar  as  our  specific  results  stated  immedi¬ 
ately  above  also  require  the  assumption  of  no  delay  and  a  noiseless  channel, 
we  have  only  gained  by  solving  this  problem  for  a  class  of  loss  functions  con¬ 
taining  the  convex,  even  loss  functions  as  a  subset.  These  results  can  be 
applied  to  the  optimum  M-level  quantisation  problem  and  the  connection  be¬ 
tween  digital  communication  and  quantisation  elucidated  by  means  of  the  reali¬ 
sation  that  the  distribution  of  the  random  variable  we  wish  to  quantise  is 
equivalent  to  the  conditional  distribution  of  the  input  process  conditioned  upon 
the  past  of  the  transmitted  sequence;  the  two  problems  are  very  similar,  al¬ 
though  the  digital  communication  problem  is  somewhat  more  general. 
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An  extenfion  to  our  work  in  different  direction  involves  the  considera¬ 
tion  of  the  design  problem  for  a  continuous  parameter  input  stochastic  process 
or  the  attempt  to  transmit  "analog”  waveforms  digitally  by  sampling  them  at 
times  |t^|  and  quantizing  the  sample  values.  The  optimization  problem  may 
then  be  formulated  with  the  same  system  error  criterion  as  used  previously  or 
with  a  new  one  given  by  the  time-weighted  average  of  the  previously  employed 
instantaneous  criterion;  the  time  weighted  criterion  might  take  the  form: 

y* 

System  Error  s  \  w(t,  t^)E  [/ (I(t)  -  R(t)  )  |s^]  dt  for  w>  0.  (7) 

t 

n 

If  we  maintain  the  instantaneous  criterion^  then  our  earlier  results 
generalize  to  cover  the  possibility  of  an  analog  input  in  a  direct  manner. 
Stating  these  readily  verified  results  for  the  binary  communication  system, 
we  have: 

Transmitter:  E(/(L(t^)  -  R(t„)  )  |  e^^  =  1  ,  s^  ^  )  = 

=  E  ((^(L(t^)  -  R(t^)  )  I  ,  Vi  )  (8) 

Receiver:  E  (I(t)  -  R(t))[s^'  )  =  0  for  <  ^a+1 

If  the  channel  is  noiseless,  and  4  1*  ii*  class  9  ,  then  Eq.  8  becomes  ; 
Transmitter:  2L(tJ  =  R  (t^;  s^^  =  1,  +  R  (t^ ;  *  -1  ,  (S') 

Receiver:  E  ( I(  t )  -  R ( t )  )  |  e^' )  »  0 

Notice  that  we  are  only  interested  in  L  ( t^ )  ;  it  is  only  at  time  t^  that  we 
are  required  to  transmit  in  a  manner  chosen  to  optimize  the  reconstruction  of 
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The  design  problem  with  the  time-weighted  criterion  given  in  Eq.  7  has 
thus  far  eluded  successful  analysis.  In  our  work  to  date  the  transmission  prob¬ 
lem  has  in  part  meant  the  subdivision  of  a  one  (for  noiseless  channel  trans¬ 
mission  without  delay  in  reconstruction)  or  fixiite  dimensional  space  into  two 
(for  binary  transmission)  or  more  regions.  Now  we  are  required  to  attempt 
the  solution  of  this  subdivision  problem  in  an  infinite  dimensional  function 
space,  and  it  is  much  more  difficult.  Before,  closing,  let  us  point  out  that 
insofar  as  the  transmitter  design  is  concerned,  the  condition  that  we  select 
the  transmitted  state  so  as  to  Select  in  turn  the  receiver  output  that  minimises 
Eq.  7  is  one  which  is  capable  of  direct  implementation  for  any  given  input  sig¬ 
nal;  one  just  evaluates  the  error  term  given  by  Eq.  7  for  the  given  input  and 
the  two  possible  receiver  outputs  and  selects  that  receiver  output  yielding  the 
least  error.  It  is  only  the  problem  of  an  initial  categorization  of  aU  inputs 
into  one  of  two  sets  that  is  difficult,  due  to  the  infinite  dimensionality  of  the 
relevant  function  space  for  the  input  process  sample  functions. 
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APPENDIX 

A  SIMUU^TION  EXAMPLE 


If  we  wish  to  construct  an  optimum  binary  commtmication  system  for  a 
noiseless  channel  and  a  quadratic  loss  function  given  J  independent  sample 
sequences  of  the  input  process  must  be  able  to  estimate  the 

optimum  feedback  loop  and  receiver  in  the  equivalent  delta  modulation  system. 
Our  earlier  work  indicates  that  we  must  form  estimators  for  the  quantities 
below  and  then  solve  the  simultaneous  equations;  we  assume  that  this  solution 
is  unique,  and,  if  not,  that  a  search  technique  will  inform  us  of  this. 


I'n'Vl 


(1) 

The  maximum  likelihood  and  least  mean-square  estimator  for  the  conditional 
expectation  resulting  in  the  receiver  output  is  given  by  the  arithmetic  average 
of  those  sample  sequences  at  the  time  in  question  whose  previous  histories 
satisfied  the  conditioning  on  the  past.  The  equation  for  L^  can  also  be  put 
in  a  form  suitable  for  estimation  by  using  the  receiver  estimates.  Thus,  we 
have: 


j 


(2) 


For  clarity  of  illustration,  let  us  take  a  very  specific  example  in  which 
our  dictionary  consists  of  four  sequences,  each  three  members  long.  The 
specific  sequences  are  given  by 

S.(l,3,-2)  ;  S2  =  (-4,-2;1)  ;  S3  =  (0,l,-2)  ;  S.  =  (-l,0,2) 

3  *  (3) 
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The  sketch  below  is  e  plot  of  these  sequences. 


At  Tj  ws  have  no  pest  infonnmtion,  so  that  is  fbuad  as  the  aver- 
ago  of  those  above  plus  the  average  of  those  below  divided  by  two,  where  in 
the  average  aU  the  sequences  are  used.  We  assume  that  the  Cj  lies  between 
83  and  .  Then  we  have: 

Ave.  Above  ■  (^)  ;  Ave.  Below  ■  ( -  3.  0)  ;  Ave.  of  Both  1.25)  (4) 

The  trial  was  between  sero  and  negative  unity,  and  is  above  the  cheek 
value;  therefore,  we  now  move  down  to  the  next  interval  between  adjacent 
samples,  which  is  between  83  and  8^  .  Repeating  the  above,  we  have: 

Ave.  Above  ■  ( 0 )  ;  Ave.  Below  ■  ( ^  5 )  ;  Ave.  of  Both  ■  (  -  2.  5 )  .  (5) 

The  new  cheek  value  of  Lj  ,  (  -  2.5 ),  fklls  in  the  range  of  the  trial  value 
which  was  from  (-4)  to  (’2.5)  and  thus  is  the  eatimated  equals 

(■  2)  . 
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The  receiver  output  corresponding  to  s  received  plus  one, 
would  then  be  given  by  the  sversge  above  or  ( 0 )  and  the  receiver  output 
corresponding  to  a  minus  one,  =  ~  M  >  would  be  the  average  below 

or  ( -  5 )  . 

Let  us  now  assume  that  a  plus  one  was  transmitted  (  s ^  s  1  )  and  we 
are  now  at  •  The  only  sequence  that  is  to  be  omitted  is  $2  ,  which  fell 
below  ^2  Thus,  at  this  time  the  only  remaining  sample  values  that  are 

acceptable,  because  of  conditioning  on  the  past,  are  (3,1,0)  .  Let  us 

A 

assume  that  the  trial  L2  (  ~  ^  between  one  and  three,  or  between 

and  ,  and  calculate  as  above: 

Ave.  Above  s  (3)  ;  Ave.  Below  =  (j)  ;  Ave.  Both  =  (l^)  (6) 

The  check  value  falls  into  the  same  interval  between  samples  as  the  trial  value 

and,  therefore,  i^2  ^  ^ )  equals  1  ^  .  The  receiver  output  corresponding  then 

A 

to  (  s^  =  1  ,  82  =  1 )  is  given  by  the  average  of  those  above  ~  M 

A 

as  there  is  only  one  sample  R2(  Sj  =  1  ,  S2  =  1  )  equals  3.  Similarly,  the 

A 

receiver  output  for  a  transmitted  chain  (  s^  =  1  ,  S2  =  -  1  )  ,  R2(  s^  =  1>*2~ 
would  equal  ^  > 

The  procedure  indicated  above  must  then  be  carried  out  for  all  possible 
transmitted  binary  sequences  of  length  equal  to  the  length  of  the  sequences 
being  used  (in  our  case  3).  In  general,  at  the  nth  sampling  time  we  will  have 
to  calculate  2*^  ^  £  and  2*^  receiver  outputs,  R  . 

Finally,  if  we  desire  to  determine  the  error  that  this  design  would 
lead  to  in  an  actual  application,  we  can  proceed  by  taking  additional  samples 
of  the  input  and  passing  them  through  the  system  just  built,  and  determine 
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the  reiulting  mean-squared  errors.  The  average  of  these  errors  would  yield 
an  estimate  for  the  operating  error  of  the  estimated  optimum  delta  modulation 
system. 

The  above  error  includes  both  the  estimation  error  and  the  quantization 
error  of  normal  delta  modulation.  While  this  is  the  desirable  error  with  re¬ 
gard  to  applications,  we  are  still  interested  in  determining  the  estimation 
error  alone  as  a  guide  to  the  number  of  samples  we  would  require.  If  we 
started  out  with  N  independent  sample  runs,  we  would  tend  to  reduce  the 
number  available  at  the  nth  step  by  approximately  a  factor  of  2^  and  could 
expect  to  have  only  N2  acceptable  samples  to  use  in  estimating  receiver 
and  feedback  loop  outputs.  However,  since  in  any  application  we  would  know 
the  exact  number  of  samples  left,  let  us  assume  that  there  are  J  samples 
that  terminate  above  the  level  set  and  K  samples  that  terminate  below,  and 
we  wish  to  estimate  the  error  variance  in  the  estimation  of  receiver  output. 

In  particular,  if  we  are  interested  in  the  upper  reconstruction,  we  are  con¬ 
cerned  with  the  J  samples.  The  expression  for  the  estimation  error  variance, 
£  ,  is  given  by 


where  in  Eq.  7  we  use  the  unbiased  estimator  given  by  Eq.  2.  Expansion  of 
Eq.  7  and  algebraic  manipulation  yield 
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Now  in  Eq.  8  we  make  use  of  the  independence  of  the  samples  to  factor  the 

expected  value  of  a  product  of  them  and  use  the  observation  that  the  sample 

averages  are  identical  with  those  of  the  random  variable  representing  the  input, 

I  ,  to  find: 
n 


Equation  9  is  the  desired  result,  as 
ance  decreases  as  the  reciprocal  of 
used  in  its  calculation. 


(9) 


it  indicates  that  the  estimation  error  vari- 
the  increasing  number  of  samples,  J  , 


